An image captured by an image-capturing device is evolving from an image on a film medium into digital data.
An image on digital data can be processed more easily than an image on a film medium. For this reason, an image on digital data is processed using various image processing devices.
Super resolution processing is known as one of such processing (for example, see PLT 1).
The super resolution processing is processing for generating an image of which resolution is high (high resolution image) based on an image of which resolution is low (input image).
The super resolution technique includes the following techniques.
The first super resolution technique is a reconstruction based super resolution technique described in PLT 1.
In general, an input image is processed in a deteriorate process (for example, blur or low resolution process), so that resolution of the input image is decreased.
Therefore, in the reconstruction based super resolution technique, the deteriorate process is reconstructed for the input image, so that a high resolution image is generated.
However, a plurality of types of processes can be considered as the deteriorate process. Therefore, in the reconstruction based super resolution technique, a plurality of candidates (or solutions) of high resolution images can be calculated. Therefore, in the reconstruction based super resolution technique, constraint conditions are used to narrow down candidates (solutions) of high resolution images reconstructed.
In the reconstruction based super resolution technique, a plurality of constraint conditions can be set.
In this case, typical constraint conditions will be described.
A first constraint condition is a constraint condition based on reconstruction (hereinafter referred to as “reconstruction constraint”). More specifically, the reconstruction constraint is a constraint determined based on relationship between an input image and a reconstructed high resolution image (for example, difference between an image obtained by deteriorating a high resolution image and an input image) or an amount of processing for reconstruction.
A second constraint condition is a constraint based on appropriateness of the high resolution image itself. More specifically, this constraint condition is a constraint based on regularity of a matrix of a high resolution image (this constraint condition will be hereinafter referred to as “regularization constraint”). For example, in the reconstruction based super resolution technique, a regularization term is added based on Bayes' theorem (prior probability). In this case, the regularization constraint is a constraint determined based on the added regularization term.
Then, in the reconstruction based super resolution technique, candidates (solutions) of high resolution images are narrowed down based on the reconstruction constraint and the regularization constraint. In the reconstruction based super resolution technique, each constraint is calculated as a cost in actual processing, so that the determination may be performed so as to minimize the total cost.
A second super resolution technique is a learning based super resolution technique. In the learning based super resolution technique, a dictionary is generated in advance based on relationship between a learning high resolution image and a low resolution image. The learning based super resolution technique is a technique for generating a high resolution image from an input image by using the dictionary. In this case, the dictionary may be referred to as previous knowledge. Therefore, the learning based super resolution technique may be described as employing previous knowledge as a constraint condition of a solution.
In the reconstruction based super resolution technique, a reconstruction constraint calculated using an input image is used. On the other hand, in the learning based super resolution technique, a dictionary generated based on relationship between a learning high resolution image and a low resolution image is used. The image processing device can also generate a super resolution image by replacing the reconstruction constraint of the reconstruction based super resolution technique with processing based on the dictionary of the learning based super resolution technique, and by using a regularization constraint of the reconstruction based super resolution technique. Therefore, in the description below, unless otherwise specified, the reconstruction based super resolution technique is described as a super resolution technique including the learning based super resolution technique. Unless otherwise specified, the reconstruction constraint can be replaced with processing based on the dictionary as necessary.
FIG. 8 is a block diagram illustrating an example of a configuration of an image processing device 90 restoring a general super resolution image by using the technique described in PLT 1.
The image processing device 90 includes a reconstruction constraint calculation unit 910, a regularization term calculation unit 920, and an image restoring unit 930.
The reconstruction constraint calculation unit 910 calculates a reconstruction constraint, that is, a cost of reconstruction.
The regularization term calculation unit 920 calculates a constraint based on regularization (regularization constraint), that is, a cost based on regularization.
The regularization term calculation unit 920 of the image processing device 90 using the super resolution technique described in PLT 1 uses, for example, TV (Total Variation) as regularization (for example, see NPL 1). The TV is a method of regularization for minimizing summation of absolute values of differences of pixel values between adjacent pixels.
Alternatively, the regularization term calculation unit 920 may use BTV (Bilateral Total Variation) (for example, see NPL 2). The BTV is a method of regularization for minimizing summation value of absolute values of differences of pixel values between not only adjacent pixels but pixels in the vicinity. However, in the BTV, the summation is derived after an absolute value of the difference is multiplied by an attenuation coefficient based on a position of a pixel.
The image restoring unit 930 generates (restores), as a restored image, a super resolution image for minimizing summation of the cost of reconstruction and the cost of regularization.
As described above, the image processing device 90 using a general super resolution technique restores a high resolution image of which resolution is increased in view of the reconstruction constraint and the regularization constraint.